Open three remote Zarr stores and put them in the cases dictionary.
In [2]:
case_paths={"GLOBAL":"cobalt.GLOBAL02_CAM_ZARR/","GLOBAL_MAY_NAO":"cobalt.GLOBAL02.MAY.NAO_CAM_ZARR/","GLOBAL_MAY_PAO":"cobalt.GLOBAL02.MAY.PAO_CAM_ZARR/",}cases={}# Initialize the S3 file systems3=s3fs.S3FileSystem()fork,vincase_paths.items():s3_path='s3://data-viz-server/{}'.format(v)store=s3fs.S3Map(root=s3_path,s3=s3,check=False)# Read Zarr file and put into cases dictionarycases[k]=xr.open_zarr(store=store,consolidated=True)
Each of the Zarr stores contains 114 variables indexed over latitude, longitude, and time. Here's their display() output. Notice that the first one has more time coordinates than the other two -- the time axis extends five years longer.
In [3]:
forcincases.values():display(c)
<xarray.Dataset>
Dimensions: (ilev: 27, lat: 192, lev: 26, lon: 288, nbnd: 2, slat: 191, slon: 288, time: 1033)
Coordinates:
* ilev (ilev) float64 2.194 4.895 9.882 18.05 ... 956.0 985.1 1e+03
* lat (lat) float64 -90.0 -89.06 -88.12 -87.17 ... 88.12 89.06 90.0
* lev (lev) float64 3.545 7.389 13.97 23.94 ... 929.6 970.6 992.6
* lon (lon) float64 0.0 1.25 2.5 3.75 ... 355.0 356.2 357.5 358.8
* slat (slat) float64 -89.53 -88.59 -87.64 ... 87.64 88.59 89.53
* slon (slon) float64 -0.625 0.625 1.875 3.125 ... 355.6 356.9 358.1
* time (time) object 2000-06-01 00:00:00 ... 2086-06-01 00:00:00
Dimensions without coordinates: nbnd
Data variables:
AEROD_v (time, lat, lon) float32 dask.array<chunksize=(1, 192, 288), meta=np.ndarray>
CLDHGH (time, lat, lon) float32 dask.array<chunksize=(1, 192, 288), meta=np.ndarray>
CLDICE (time, lev, lat, lon) float32 dask.array<chunksize=(1, 26, 192, 288), meta=np.ndarray>
CLDLIQ (time, lev, lat, lon) float32 dask.array<chunksize=(1, 26, 192, 288), meta=np.ndarray>
CLDLOW (time, lat, lon) float32 dask.array<chunksize=(1, 192, 288), meta=np.ndarray>
CLDMED (time, lat, lon) float32 dask.array<chunksize=(1, 192, 288), meta=np.ndarray>
CLDTOT (time, lat, lon) float32 dask.array<chunksize=(1, 192, 288), meta=np.ndarray>
CLOUD (time, lev, lat, lon) float32 dask.array<chunksize=(1, 26, 192, 288), meta=np.ndarray>
CONCLD (time, lev, lat, lon) float32 dask.array<chunksize=(1, 26, 192, 288), meta=np.ndarray>
DCQ (time, lev, lat, lon) float32 dask.array<chunksize=(1, 26, 192, 288), meta=np.ndarray>
DTCOND (time, lev, lat, lon) float32 dask.array<chunksize=(1, 26, 192, 288), meta=np.ndarray>
DTV (time, lev, lat, lon) float32 dask.array<chunksize=(1, 26, 192, 288), meta=np.ndarray>
EMIS (time, lev, lat, lon) float32 dask.array<chunksize=(1, 26, 192, 288), meta=np.ndarray>
FICE (time, lev, lat, lon) float32 dask.array<chunksize=(1, 26, 192, 288), meta=np.ndarray>
FLDS (time, lat, lon) float32 dask.array<chunksize=(1, 192, 288), meta=np.ndarray>
FLDSC (time, lat, lon) float32 dask.array<chunksize=(1, 192, 288), meta=np.ndarray>
FLNS (time, lat, lon) float32 dask.array<chunksize=(1, 192, 288), meta=np.ndarray>
FLNSC (time, lat, lon) float32 dask.array<chunksize=(1, 192, 288), meta=np.ndarray>
FLNT (time, lat, lon) float32 dask.array<chunksize=(1, 192, 288), meta=np.ndarray>
FLNTC (time, lat, lon) float32 dask.array<chunksize=(1, 192, 288), meta=np.ndarray>
FLUT (time, lat, lon) float32 dask.array<chunksize=(1, 192, 288), meta=np.ndarray>
FLUTC (time, lat, lon) float32 dask.array<chunksize=(1, 192, 288), meta=np.ndarray>
FSDS (time, lat, lon) float32 dask.array<chunksize=(1, 192, 288), meta=np.ndarray>
FSDSC (time, lat, lon) float32 dask.array<chunksize=(1, 192, 288), meta=np.ndarray>
FSDTOA (time, lat, lon) float32 dask.array<chunksize=(1, 192, 288), meta=np.ndarray>
FSNS (time, lat, lon) float32 dask.array<chunksize=(1, 192, 288), meta=np.ndarray>
FSNSC (time, lat, lon) float32 dask.array<chunksize=(1, 192, 288), meta=np.ndarray>
FSNT (time, lat, lon) float32 dask.array<chunksize=(1, 192, 288), meta=np.ndarray>
FSNTC (time, lat, lon) float32 dask.array<chunksize=(1, 192, 288), meta=np.ndarray>
FSNTOA (time, lat, lon) float32 dask.array<chunksize=(1, 192, 288), meta=np.ndarray>
FSNTOAC (time, lat, lon) float32 dask.array<chunksize=(1, 192, 288), meta=np.ndarray>
FSUTOA (time, lat, lon) float32 dask.array<chunksize=(1, 192, 288), meta=np.ndarray>
ICEFRAC (time, lat, lon) float32 dask.array<chunksize=(1, 192, 288), meta=np.ndarray>
ICIMR (time, lev, lat, lon) float32 dask.array<chunksize=(1, 26, 192, 288), meta=np.ndarray>
ICWMR (time, lev, lat, lon) float32 dask.array<chunksize=(1, 26, 192, 288), meta=np.ndarray>
LANDFRAC (time, lat, lon) float32 dask.array<chunksize=(1, 192, 288), meta=np.ndarray>
LHFLX (time, lat, lon) float32 dask.array<chunksize=(1, 192, 288), meta=np.ndarray>
LWCF (time, lat, lon) float32 dask.array<chunksize=(1, 192, 288), meta=np.ndarray>
OCNFRAC (time, lat, lon) float32 dask.array<chunksize=(1, 192, 288), meta=np.ndarray>
OMEGA (time, lev, lat, lon) float32 dask.array<chunksize=(1, 26, 192, 288), meta=np.ndarray>
OMEGAT (time, lev, lat, lon) float32 dask.array<chunksize=(1, 26, 192, 288), meta=np.ndarray>
P0 (time) float64 dask.array<chunksize=(1033,), meta=np.ndarray>
PBLH (time, lat, lon) float32 dask.array<chunksize=(1, 192, 288), meta=np.ndarray>
PHIS (time, lat, lon) float32 dask.array<chunksize=(1, 192, 288), meta=np.ndarray>
PRECC (time, lat, lon) float32 dask.array<chunksize=(1, 192, 288), meta=np.ndarray>
PRECL (time, lat, lon) float32 dask.array<chunksize=(1, 192, 288), meta=np.ndarray>
PRECSC (time, lat, lon) float32 dask.array<chunksize=(1, 192, 288), meta=np.ndarray>
PRECSL (time, lat, lon) float32 dask.array<chunksize=(1, 192, 288), meta=np.ndarray>
PS (time, lat, lon) float32 dask.array<chunksize=(1, 192, 288), meta=np.ndarray>
PSL (time, lat, lon) float32 dask.array<chunksize=(1, 192, 288), meta=np.ndarray>
Q (time, lev, lat, lon) float32 dask.array<chunksize=(1, 26, 192, 288), meta=np.ndarray>
QFLX (time, lat, lon) float32 dask.array<chunksize=(1, 192, 288), meta=np.ndarray>
QREFHT (time, lat, lon) float32 dask.array<chunksize=(1, 192, 288), meta=np.ndarray>
QRL (time, lev, lat, lon) float32 dask.array<chunksize=(1, 26, 192, 288), meta=np.ndarray>
QRS (time, lev, lat, lon) float32 dask.array<chunksize=(1, 26, 192, 288), meta=np.ndarray>
RELHUM (time, lev, lat, lon) float32 dask.array<chunksize=(1, 26, 192, 288), meta=np.ndarray>
SFCLDICE (time, lat, lon) float32 dask.array<chunksize=(1, 192, 288), meta=np.ndarray>
SFCLDLIQ (time, lat, lon) float32 dask.array<chunksize=(1, 192, 288), meta=np.ndarray>
SHFLX (time, lat, lon) float32 dask.array<chunksize=(1, 192, 288), meta=np.ndarray>
SNOWHICE (time, lat, lon) float32 dask.array<chunksize=(1, 192, 288), meta=np.ndarray>
SNOWHLND (time, lat, lon) float32 dask.array<chunksize=(1, 192, 288), meta=np.ndarray>
SOLIN (time, lat, lon) float32 dask.array<chunksize=(1, 192, 288), meta=np.ndarray>
SWCF (time, lat, lon) float32 dask.array<chunksize=(1, 192, 288), meta=np.ndarray>
T (time, lev, lat, lon) float32 dask.array<chunksize=(1, 26, 192, 288), meta=np.ndarray>
TAUX (time, lat, lon) float32 dask.array<chunksize=(1, 192, 288), meta=np.ndarray>
TAUY (time, lat, lon) float32 dask.array<chunksize=(1, 192, 288), meta=np.ndarray>
TGCLDCWP (time, lat, lon) float32 dask.array<chunksize=(1, 192, 288), meta=np.ndarray>
TGCLDIWP (time, lat, lon) float32 dask.array<chunksize=(1, 192, 288), meta=np.ndarray>
TGCLDLWP (time, lat, lon) float32 dask.array<chunksize=(1, 192, 288), meta=np.ndarray>
TMQ (time, lat, lon) float32 dask.array<chunksize=(1, 192, 288), meta=np.ndarray>
TREFHT (time, lat, lon) float32 dask.array<chunksize=(1, 192, 288), meta=np.ndarray>
TS (time, lat, lon) float32 dask.array<chunksize=(1, 192, 288), meta=np.ndarray>
TSMN (time, lat, lon) float32 dask.array<chunksize=(1, 192, 288), meta=np.ndarray>
TSMX (time, lat, lon) float32 dask.array<chunksize=(1, 192, 288), meta=np.ndarray>
U (time, lev, lat, lon) float32 dask.array<chunksize=(1, 26, 192, 288), meta=np.ndarray>
U10 (time, lat, lon) float32 dask.array<chunksize=(1, 192, 288), meta=np.ndarray>
UU (time, lev, lat, lon) float32 dask.array<chunksize=(1, 26, 192, 288), meta=np.ndarray>
V (time, lev, lat, lon) float32 dask.array<chunksize=(1, 26, 192, 288), meta=np.ndarray>
VD01 (time, lev, lat, lon) float32 dask.array<chunksize=(1, 26, 192, 288), meta=np.ndarray>
VQ (time, lev, lat, lon) float32 dask.array<chunksize=(1, 26, 192, 288), meta=np.ndarray>
VT (time, lev, lat, lon) float32 dask.array<chunksize=(1, 26, 192, 288), meta=np.ndarray>
VU (time, lev, lat, lon) float32 dask.array<chunksize=(1, 26, 192, 288), meta=np.ndarray>
VV (time, lev, lat, lon) float32 dask.array<chunksize=(1, 26, 192, 288), meta=np.ndarray>
Z3 (time, lev, lat, lon) float32 dask.array<chunksize=(1, 26, 192, 288), meta=np.ndarray>
ch4vmr (time) float64 dask.array<chunksize=(1,), meta=np.ndarray>
co2vmr (time) float64 dask.array<chunksize=(1,), meta=np.ndarray>
date (time) int32 dask.array<chunksize=(1,), meta=np.ndarray>
date_written (time) |S8 dask.array<chunksize=(1,), meta=np.ndarray>
datesec (time) int32 dask.array<chunksize=(1,), meta=np.ndarray>
f11vmr (time) float64 dask.array<chunksize=(1,), meta=np.ndarray>
f12vmr (time) float64 dask.array<chunksize=(1,), meta=np.ndarray>
gw (time, lat) float64 dask.array<chunksize=(1, 192), meta=np.ndarray>
hyai (time, ilev) float64 dask.array<chunksize=(1, 27), meta=np.ndarray>
hyam (time, lev) float64 dask.array<chunksize=(1, 26), meta=np.ndarray>
hybi (time, ilev) float64 dask.array<chunksize=(1, 27), meta=np.ndarray>
hybm (time, lev) float64 dask.array<chunksize=(1, 26), meta=np.ndarray>
mdt (time) int32 dask.array<chunksize=(1033,), meta=np.ndarray>
n2ovmr (time) float64 dask.array<chunksize=(1,), meta=np.ndarray>
nbdate (time) int32 dask.array<chunksize=(1033,), meta=np.ndarray>
nbsec (time) int32 dask.array<chunksize=(1033,), meta=np.ndarray>
ndbase (time) int32 dask.array<chunksize=(1033,), meta=np.ndarray>
ndcur (time) int32 dask.array<chunksize=(1,), meta=np.ndarray>
nlon (time, lat) int32 dask.array<chunksize=(1, 192), meta=np.ndarray>
nsbase (time) int32 dask.array<chunksize=(1033,), meta=np.ndarray>
nscur (time) int32 dask.array<chunksize=(1,), meta=np.ndarray>
nsteph (time) int32 dask.array<chunksize=(1,), meta=np.ndarray>
ntrk (time) int32 dask.array<chunksize=(1033,), meta=np.ndarray>
ntrm (time) int32 dask.array<chunksize=(1033,), meta=np.ndarray>
ntrn (time) int32 dask.array<chunksize=(1033,), meta=np.ndarray>
sol_tsi (time) float64 dask.array<chunksize=(1,), meta=np.ndarray>
time_bnds (time, nbnd) object dask.array<chunksize=(1033, 2), meta=np.ndarray>
time_written (time) |S8 dask.array<chunksize=(1,), meta=np.ndarray>
w_stag (time, slat) float64 dask.array<chunksize=(1, 191), meta=np.ndarray>
wnummax (time, lat) int32 dask.array<chunksize=(1, 192), meta=np.ndarray>
Attributes:
Conventions: CF-1.0
Version: $Name$
case: f09_g16.B.cobalt.GLOBAL02
host: n716022
initial_file: f09_g16.B.04.EXT.cam.i.2077-05-01-00000.nc
logname: clmfrm01
revision_Id: $Id$
source: CAM
title: UNSET
topography_file: /e/04/clmfrm/din/atm/cam/topo/USGS-gtopo30_0.9x1.25_rem...
Concatenating these datasets works fine and implicitly uses the NaN fillvalue.
In [4]:
# concatenate dataset along new Index dim using dictionary keysdata=xr.concat(objs=cases.values(),dim=pd.Index(list(cases.keys()),name='ao_phase',dtype='U15'))data
Out[4]:
<xarray.Dataset>
Dimensions: (ao_phase: 3, ilev: 27, lat: 192, lev: 26, lon: 288, nbnd: 2, slat: 191, slon: 288, time: 1033)
Coordinates:
* ilev (ilev) float64 2.194 4.895 9.882 18.05 ... 956.0 985.1 1e+03
* slat (slat) float64 -89.53 -88.59 -87.64 ... 87.64 88.59 89.53
* lat (lat) float64 -90.0 -89.06 -88.12 -87.17 ... 88.12 89.06 90.0
* slon (slon) float64 -0.625 0.625 1.875 3.125 ... 355.6 356.9 358.1
* lon (lon) float64 0.0 1.25 2.5 3.75 ... 355.0 356.2 357.5 358.8
* time (time) object 2000-06-01 00:00:00 ... 2086-06-01 00:00:00
* lev (lev) float64 3.545 7.389 13.97 23.94 ... 929.6 970.6 992.6
* ao_phase (ao_phase) object 'GLOBAL' 'GLOBAL_MAY_NAO' 'GLOBAL_MAY_PAO'
Dimensions without coordinates: nbnd
Data variables:
AEROD_v (ao_phase, time, lat, lon) float32 dask.array<chunksize=(1, 1, 192, 288), meta=np.ndarray>
CLDHGH (ao_phase, time, lat, lon) float32 dask.array<chunksize=(1, 1, 192, 288), meta=np.ndarray>
CLDICE (ao_phase, time, lev, lat, lon) float32 dask.array<chunksize=(1, 1, 26, 192, 288), meta=np.ndarray>
CLDLIQ (ao_phase, time, lev, lat, lon) float32 dask.array<chunksize=(1, 1, 26, 192, 288), meta=np.ndarray>
CLDLOW (ao_phase, time, lat, lon) float32 dask.array<chunksize=(1, 1, 192, 288), meta=np.ndarray>
CLDMED (ao_phase, time, lat, lon) float32 dask.array<chunksize=(1, 1, 192, 288), meta=np.ndarray>
CLDTOT (ao_phase, time, lat, lon) float32 dask.array<chunksize=(1, 1, 192, 288), meta=np.ndarray>
CLOUD (ao_phase, time, lev, lat, lon) float32 dask.array<chunksize=(1, 1, 26, 192, 288), meta=np.ndarray>
CONCLD (ao_phase, time, lev, lat, lon) float32 dask.array<chunksize=(1, 1, 26, 192, 288), meta=np.ndarray>
DCQ (ao_phase, time, lev, lat, lon) float32 dask.array<chunksize=(1, 1, 26, 192, 288), meta=np.ndarray>
DTCOND (ao_phase, time, lev, lat, lon) float32 dask.array<chunksize=(1, 1, 26, 192, 288), meta=np.ndarray>
DTV (ao_phase, time, lev, lat, lon) float32 dask.array<chunksize=(1, 1, 26, 192, 288), meta=np.ndarray>
EMIS (ao_phase, time, lev, lat, lon) float32 dask.array<chunksize=(1, 1, 26, 192, 288), meta=np.ndarray>
FICE (ao_phase, time, lev, lat, lon) float32 dask.array<chunksize=(1, 1, 26, 192, 288), meta=np.ndarray>
FLDS (ao_phase, time, lat, lon) float32 dask.array<chunksize=(1, 1, 192, 288), meta=np.ndarray>
FLDSC (ao_phase, time, lat, lon) float32 dask.array<chunksize=(1, 1, 192, 288), meta=np.ndarray>
FLNS (ao_phase, time, lat, lon) float32 dask.array<chunksize=(1, 1, 192, 288), meta=np.ndarray>
FLNSC (ao_phase, time, lat, lon) float32 dask.array<chunksize=(1, 1, 192, 288), meta=np.ndarray>
FLNT (ao_phase, time, lat, lon) float32 dask.array<chunksize=(1, 1, 192, 288), meta=np.ndarray>
FLNTC (ao_phase, time, lat, lon) float32 dask.array<chunksize=(1, 1, 192, 288), meta=np.ndarray>
FLUT (ao_phase, time, lat, lon) float32 dask.array<chunksize=(1, 1, 192, 288), meta=np.ndarray>
FLUTC (ao_phase, time, lat, lon) float32 dask.array<chunksize=(1, 1, 192, 288), meta=np.ndarray>
FSDS (ao_phase, time, lat, lon) float32 dask.array<chunksize=(1, 1, 192, 288), meta=np.ndarray>
FSDSC (ao_phase, time, lat, lon) float32 dask.array<chunksize=(1, 1, 192, 288), meta=np.ndarray>
FSDTOA (ao_phase, time, lat, lon) float32 dask.array<chunksize=(1, 1, 192, 288), meta=np.ndarray>
FSNS (ao_phase, time, lat, lon) float32 dask.array<chunksize=(1, 1, 192, 288), meta=np.ndarray>
FSNSC (ao_phase, time, lat, lon) float32 dask.array<chunksize=(1, 1, 192, 288), meta=np.ndarray>
FSNT (ao_phase, time, lat, lon) float32 dask.array<chunksize=(1, 1, 192, 288), meta=np.ndarray>
FSNTC (ao_phase, time, lat, lon) float32 dask.array<chunksize=(1, 1, 192, 288), meta=np.ndarray>
FSNTOA (ao_phase, time, lat, lon) float32 dask.array<chunksize=(1, 1, 192, 288), meta=np.ndarray>
FSNTOAC (ao_phase, time, lat, lon) float32 dask.array<chunksize=(1, 1, 192, 288), meta=np.ndarray>
FSUTOA (ao_phase, time, lat, lon) float32 dask.array<chunksize=(1, 1, 192, 288), meta=np.ndarray>
ICEFRAC (ao_phase, time, lat, lon) float32 dask.array<chunksize=(1, 1, 192, 288), meta=np.ndarray>
ICIMR (ao_phase, time, lev, lat, lon) float32 dask.array<chunksize=(1, 1, 26, 192, 288), meta=np.ndarray>
ICWMR (ao_phase, time, lev, lat, lon) float32 dask.array<chunksize=(1, 1, 26, 192, 288), meta=np.ndarray>
LANDFRAC (ao_phase, time, lat, lon) float32 dask.array<chunksize=(1, 1, 192, 288), meta=np.ndarray>
LHFLX (ao_phase, time, lat, lon) float32 dask.array<chunksize=(1, 1, 192, 288), meta=np.ndarray>
LWCF (ao_phase, time, lat, lon) float32 dask.array<chunksize=(1, 1, 192, 288), meta=np.ndarray>
OCNFRAC (ao_phase, time, lat, lon) float32 dask.array<chunksize=(1, 1, 192, 288), meta=np.ndarray>
OMEGA (ao_phase, time, lev, lat, lon) float32 dask.array<chunksize=(1, 1, 26, 192, 288), meta=np.ndarray>
OMEGAT (ao_phase, time, lev, lat, lon) float32 dask.array<chunksize=(1, 1, 26, 192, 288), meta=np.ndarray>
P0 (ao_phase, time) float64 dask.array<chunksize=(1, 1033), meta=np.ndarray>
PBLH (ao_phase, time, lat, lon) float32 dask.array<chunksize=(1, 1, 192, 288), meta=np.ndarray>
PHIS (ao_phase, time, lat, lon) float32 dask.array<chunksize=(1, 1, 192, 288), meta=np.ndarray>
PRECC (ao_phase, time, lat, lon) float32 dask.array<chunksize=(1, 1, 192, 288), meta=np.ndarray>
PRECL (ao_phase, time, lat, lon) float32 dask.array<chunksize=(1, 1, 192, 288), meta=np.ndarray>
PRECSC (ao_phase, time, lat, lon) float32 dask.array<chunksize=(1, 1, 192, 288), meta=np.ndarray>
PRECSL (ao_phase, time, lat, lon) float32 dask.array<chunksize=(1, 1, 192, 288), meta=np.ndarray>
PS (ao_phase, time, lat, lon) float32 dask.array<chunksize=(1, 1, 192, 288), meta=np.ndarray>
PSL (ao_phase, time, lat, lon) float32 dask.array<chunksize=(1, 1, 192, 288), meta=np.ndarray>
Q (ao_phase, time, lev, lat, lon) float32 dask.array<chunksize=(1, 1, 26, 192, 288), meta=np.ndarray>
QFLX (ao_phase, time, lat, lon) float32 dask.array<chunksize=(1, 1, 192, 288), meta=np.ndarray>
QREFHT (ao_phase, time, lat, lon) float32 dask.array<chunksize=(1, 1, 192, 288), meta=np.ndarray>
QRL (ao_phase, time, lev, lat, lon) float32 dask.array<chunksize=(1, 1, 26, 192, 288), meta=np.ndarray>
QRS (ao_phase, time, lev, lat, lon) float32 dask.array<chunksize=(1, 1, 26, 192, 288), meta=np.ndarray>
RELHUM (ao_phase, time, lev, lat, lon) float32 dask.array<chunksize=(1, 1, 26, 192, 288), meta=np.ndarray>
SFCLDICE (ao_phase, time, lat, lon) float32 dask.array<chunksize=(1, 1, 192, 288), meta=np.ndarray>
SFCLDLIQ (ao_phase, time, lat, lon) float32 dask.array<chunksize=(1, 1, 192, 288), meta=np.ndarray>
SHFLX (ao_phase, time, lat, lon) float32 dask.array<chunksize=(1, 1, 192, 288), meta=np.ndarray>
SNOWHICE (ao_phase, time, lat, lon) float32 dask.array<chunksize=(1, 1, 192, 288), meta=np.ndarray>
SNOWHLND (ao_phase, time, lat, lon) float32 dask.array<chunksize=(1, 1, 192, 288), meta=np.ndarray>
SOLIN (ao_phase, time, lat, lon) float32 dask.array<chunksize=(1, 1, 192, 288), meta=np.ndarray>
SWCF (ao_phase, time, lat, lon) float32 dask.array<chunksize=(1, 1, 192, 288), meta=np.ndarray>
T (ao_phase, time, lev, lat, lon) float32 dask.array<chunksize=(1, 1, 26, 192, 288), meta=np.ndarray>
TAUX (ao_phase, time, lat, lon) float32 dask.array<chunksize=(1, 1, 192, 288), meta=np.ndarray>
TAUY (ao_phase, time, lat, lon) float32 dask.array<chunksize=(1, 1, 192, 288), meta=np.ndarray>
TGCLDCWP (ao_phase, time, lat, lon) float32 dask.array<chunksize=(1, 1, 192, 288), meta=np.ndarray>
TGCLDIWP (ao_phase, time, lat, lon) float32 dask.array<chunksize=(1, 1, 192, 288), meta=np.ndarray>
TGCLDLWP (ao_phase, time, lat, lon) float32 dask.array<chunksize=(1, 1, 192, 288), meta=np.ndarray>
TMQ (ao_phase, time, lat, lon) float32 dask.array<chunksize=(1, 1, 192, 288), meta=np.ndarray>
TREFHT (ao_phase, time, lat, lon) float32 dask.array<chunksize=(1, 1, 192, 288), meta=np.ndarray>
TS (ao_phase, time, lat, lon) float32 dask.array<chunksize=(1, 1, 192, 288), meta=np.ndarray>
TSMN (ao_phase, time, lat, lon) float32 dask.array<chunksize=(1, 1, 192, 288), meta=np.ndarray>
TSMX (ao_phase, time, lat, lon) float32 dask.array<chunksize=(1, 1, 192, 288), meta=np.ndarray>
U (ao_phase, time, lev, lat, lon) float32 dask.array<chunksize=(1, 1, 26, 192, 288), meta=np.ndarray>
U10 (ao_phase, time, lat, lon) float32 dask.array<chunksize=(1, 1, 192, 288), meta=np.ndarray>
UU (ao_phase, time, lev, lat, lon) float32 dask.array<chunksize=(1, 1, 26, 192, 288), meta=np.ndarray>
V (ao_phase, time, lev, lat, lon) float32 dask.array<chunksize=(1, 1, 26, 192, 288), meta=np.ndarray>
VD01 (ao_phase, time, lev, lat, lon) float32 dask.array<chunksize=(1, 1, 26, 192, 288), meta=np.ndarray>
VQ (ao_phase, time, lev, lat, lon) float32 dask.array<chunksize=(1, 1, 26, 192, 288), meta=np.ndarray>
VT (ao_phase, time, lev, lat, lon) float32 dask.array<chunksize=(1, 1, 26, 192, 288), meta=np.ndarray>
VU (ao_phase, time, lev, lat, lon) float32 dask.array<chunksize=(1, 1, 26, 192, 288), meta=np.ndarray>
VV (ao_phase, time, lev, lat, lon) float32 dask.array<chunksize=(1, 1, 26, 192, 288), meta=np.ndarray>
Z3 (ao_phase, time, lev, lat, lon) float32 dask.array<chunksize=(1, 1, 26, 192, 288), meta=np.ndarray>
ch4vmr (ao_phase, time) float64 dask.array<chunksize=(1, 1), meta=np.ndarray>
co2vmr (ao_phase, time) float64 dask.array<chunksize=(1, 1), meta=np.ndarray>
date (ao_phase, time) float64 dask.array<chunksize=(1, 1), meta=np.ndarray>
date_written (ao_phase, time) object dask.array<chunksize=(1, 1), meta=np.ndarray>
datesec (ao_phase, time) float64 dask.array<chunksize=(1, 1), meta=np.ndarray>
f11vmr (ao_phase, time) float64 dask.array<chunksize=(1, 1), meta=np.ndarray>
f12vmr (ao_phase, time) float64 dask.array<chunksize=(1, 1), meta=np.ndarray>
gw (ao_phase, time, lat) float64 dask.array<chunksize=(1, 1, 192), meta=np.ndarray>
hyai (ao_phase, time, ilev) float64 dask.array<chunksize=(1, 1, 27), meta=np.ndarray>
hyam (ao_phase, time, lev) float64 dask.array<chunksize=(1, 1, 26), meta=np.ndarray>
hybi (ao_phase, time, ilev) float64 dask.array<chunksize=(1, 1, 27), meta=np.ndarray>
hybm (ao_phase, time, lev) float64 dask.array<chunksize=(1, 1, 26), meta=np.ndarray>
mdt (ao_phase, time) float64 dask.array<chunksize=(1, 1033), meta=np.ndarray>
n2ovmr (ao_phase, time) float64 dask.array<chunksize=(1, 1), meta=np.ndarray>
nbdate (ao_phase, time) float64 dask.array<chunksize=(1, 1033), meta=np.ndarray>
nbsec (ao_phase, time) float64 dask.array<chunksize=(1, 1033), meta=np.ndarray>
ndbase (ao_phase, time) float64 dask.array<chunksize=(1, 1033), meta=np.ndarray>
ndcur (ao_phase, time) float64 dask.array<chunksize=(1, 1), meta=np.ndarray>
nlon (ao_phase, time, lat) float64 dask.array<chunksize=(1, 1, 192), meta=np.ndarray>
nsbase (ao_phase, time) float64 dask.array<chunksize=(1, 1033), meta=np.ndarray>
nscur (ao_phase, time) float64 dask.array<chunksize=(1, 1), meta=np.ndarray>
nsteph (ao_phase, time) float64 dask.array<chunksize=(1, 1), meta=np.ndarray>
ntrk (ao_phase, time) float64 dask.array<chunksize=(1, 1033), meta=np.ndarray>
ntrm (ao_phase, time) float64 dask.array<chunksize=(1, 1033), meta=np.ndarray>
ntrn (ao_phase, time) float64 dask.array<chunksize=(1, 1033), meta=np.ndarray>
sol_tsi (ao_phase, time) float64 dask.array<chunksize=(1, 1), meta=np.ndarray>
time_bnds (ao_phase, time, nbnd) object dask.array<chunksize=(1, 1033, 2), meta=np.ndarray>
time_written (ao_phase, time) object dask.array<chunksize=(1, 1), meta=np.ndarray>
w_stag (ao_phase, time, slat) float64 dask.array<chunksize=(1, 1, 191), meta=np.ndarray>
wnummax (ao_phase, time, lat) float64 dask.array<chunksize=(1, 1, 192), meta=np.ndarray>
Attributes:
Conventions: CF-1.0
Version: $Name$
case: f09_g16.B.cobalt.GLOBAL02
host: n716022
initial_file: f09_g16.B.04.EXT.cam.i.2077-05-01-00000.nc
logname: clmfrm01
revision_Id: $Id$
source: CAM
title: UNSET
topography_file: /e/04/clmfrm/din/atm/cam/topo/USGS-gtopo30_0.9x1.25_rem...
/home/ubuntu/anaconda3/envs/bokenv/lib/python3.7/site-packages/xarray/conventions.py:188: SerializationWarning: variable None has data in the form of a dask array with dtype=object, which means it is being loaded into memory to determine a data type that can be safely stored on disk. To avoid this, coerce this variable to a fixed-size dtype with astype() before saving it.
SerializationWarning,
---------------------------------------------------------------------------TypeError Traceback (most recent call last)
<ipython-input-5-078f027b4301> in <module> 6 7with ProgressBar():----> 8zs = data.to_zarr(out_store, encoding=encoding, consolidated=True)~/anaconda3/envs/bokenv/lib/python3.7/site-packages/xarray/core/dataset.py in to_zarr(self, store, mode, synchronizer, group, encoding, compute, consolidated, append_dim) 1656 compute=compute, 1657 consolidated=consolidated,-> 1658append_dim=append_dim, 1659 )
1660~/anaconda3/envs/bokenv/lib/python3.7/site-packages/xarray/backends/api.py in to_zarr(dataset, store, mode, synchronizer, group, encoding, compute, consolidated, append_dim) 1351 writer = ArrayWriter() 1352# TODO: figure out how to properly handle unlimited_dims-> 1353dump_to_store(dataset, zstore, writer, encoding=encoding) 1354 writes = writer.sync(compute=compute) 1355~/anaconda3/envs/bokenv/lib/python3.7/site-packages/xarray/backends/api.py in dump_to_store(dataset, store, writer, encoder, encoding, unlimited_dims) 1126 variables, attrs = encoder(variables, attrs) 1127-> 1128store.store(variables, attrs, check_encoding, writer, unlimited_dims=unlimited_dims) 1129 1130~/anaconda3/envs/bokenv/lib/python3.7/site-packages/xarray/backends/zarr.py in store(self, variables, attributes, check_encoding_set, writer, unlimited_dims) 394 variables_without_encoding ={vn: variables[vn]for vn in new_variables} 395 variables_encoded, attributes = self.encode(
--> 396variables_without_encoding, attributes
397 )
398~/anaconda3/envs/bokenv/lib/python3.7/site-packages/xarray/backends/common.py in encode(self, variables, attributes) 230 231 """
--> 232variables ={k: self.encode_variable(v)for k, v in variables.items()} 233 attributes ={k: self.encode_attribute(v)for k, v in attributes.items()} 234return variables, attributes
~/anaconda3/envs/bokenv/lib/python3.7/site-packages/xarray/backends/common.py in <dictcomp>(.0) 230 231 """
--> 232variables ={k: self.encode_variable(v)for k, v in variables.items()} 233 attributes ={k: self.encode_attribute(v)for k, v in attributes.items()} 234return variables, attributes
~/anaconda3/envs/bokenv/lib/python3.7/site-packages/xarray/backends/zarr.py in encode_variable(self, variable) 351 352def encode_variable(self, variable):--> 353variable = encode_zarr_variable(variable) 354return variable
355~/anaconda3/envs/bokenv/lib/python3.7/site-packages/xarray/backends/zarr.py in encode_zarr_variable(var, needs_copy, name) 243 """
244--> 245var = conventions.encode_cf_variable(var, name=name) 246 247# zarr allows unicode, but not variable-length strings, so it's both~/anaconda3/envs/bokenv/lib/python3.7/site-packages/xarray/conventions.py in encode_cf_variable(var, needs_copy, name) 248 variables.UnsignedIntegerCoder(), 249 ]:
--> 250var = coder.encode(var, name=name) 251 252# TODO(shoyer): convert all of these to use coders, too:~/anaconda3/envs/bokenv/lib/python3.7/site-packages/xarray/coding/times.py in encode(self, variable, name) 412 ):
413 (data, units, calendar) = encode_cf_datetime(
--> 414data, encoding.pop("units",None), encoding.pop("calendar",None) 415 )
416 safe_setitem(attrs,"units", units, name=name)~/anaconda3/envs/bokenv/lib/python3.7/site-packages/xarray/coding/times.py in encode_cf_datetime(dates, units, calendar) 355 356if units isNone:--> 357units = infer_datetime_units(dates) 358else: 359 units = _cleanup_netcdf_time_units(units)~/anaconda3/envs/bokenv/lib/python3.7/site-packages/xarray/coding/times.py in infer_datetime_units(dates) 249 reference_date = dates[0]if len(dates)>0else"1970-01-01" 250 reference_date = format_cftime_datetime(reference_date)--> 251unique_timedeltas = np.unique(np.diff(dates)) 252if unique_timedeltas.dtype == np.dtype("O"): 253# Convert to np.timedelta64 objects using pandas to work around a<__array_function__ internals> in diff(*args, **kwargs)~/anaconda3/envs/bokenv/lib/python3.7/site-packages/numpy/lib/function_base.py in diff(a, n, axis, prepend, append) 1278 op = not_equal if a.dtype == np.bool_ else subtract
1279for _ in range(n):-> 1280a = op(a[slice1], a[slice2]) 1281 1282return a
TypeError: unsupported operand type(s) for -: 'float' and 'cftime._cftime.DatetimeNoLeap'
If we remove the offending times prior to Zarr'ing, everything works.
In [6]:
# find the times that are not represented across all three datasetssets=[set(x.time.values)forxincases.values()]times_to_drop=set.union(*sets)-set.intersection(*sets)times_to_drop